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The staff and principal contributors to this project are shown in 
Table 1. Numerous members of the College of Education faculty and the student 
body made unreimbursed contributions to this project. Its progress was par- 
ticularly stimulated by the presence in the same offices of the statewide 
Evaluation of the Illinois Gifted Program directed by Ernest House. CIRCE 
staff members J. Thomas Hastings and James Wardrop were frequent contributors. 
The personal assistance of Robert Hardy and his OSPI state staff was also 
instrumental in the progress of the CIRCE Gifted Experimental Project. 


The proposal for second-year funding identified four operating 
objectives or emphases to give substance to the project's global objectives. 
The work done on each emphasis is briefly mentioned in Table 2 and described 
in detail in Table 3. 


The allocation of funds and staff time to project activities are 
estimated in Table 3. The funding for many of these activities came jointly 
from the Experimental Project and other CIRCE contracts. 


Products of these activities are included in the appendix as 
follows: 


A. The Accountability Notebook* 

B. Guide for Rating an Evaluation Plan 
C. Three Case Studies* 

D. Student Questionnaires 

E. "Measuring What Students Learn'’* 

F. "National Assessment"* 

G. CIRCE Resource Center 

H. Curriculum Analysis Training--manual 


The project was evaluated internally and externally. A team con- 
sisting of Glenn Bracht (chairman), Duane Neet, and Joe Walker visited the 
project for one day and prepared the report included here just ahead of the 
appendix. 


*These appendices will be blank except in official copies of this final 
report; the Accountability Notebook will be omitted in all copies. 
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Table l 


Activities and Responsibilities of Persons Working One-Fourth Time 
or More on the CIRCE OSPI Gifted Experimental Project 


1970-71 
Percent of 
Staff Member year worked Activities and responsibilities 
Robert E. Stake (.25) Project director 


Accountability Notebook development 

Measuring student learnings 
(performance contracting) 

CMAS video~-tape 


Terry Denny (.66) Accountability Notebook development, 
follow-up 
Site visits, liaison 
Training with state staff 


Robert Wolf (.50) Classroom report, follow-up 
Accountability Notebook development, 
follow-up 
Teachers’ feelings toward evaluation, 
study 


Terry Hopkins Elofson 'ee)) CMAS video-tape, development 
Project administration 
Correspondence course, development 


Craig Gjerde Casa) Evaluation of high school gifted 
program 
Site visit for training 


Christine George Ciao) Resource center 
Accountability Notebook, resources 
Site visit for training 
Clencie Cotton (.25) Consultation for gifted projects and 
community relations 
Shadow study, junior high school 
Edward Kelly C25) Evaluation design, theory 


Mary Sue German Cres) Resource center 
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Table 1 (continued) 


Staff Member 


Percent of 
year worked 


Activities and responsibilities 


Dennis Gooler 


Gordon Hoke 


Consultants: 


Les McLean 

Don Cunningham 
Richard Mann 
Gene Jabker 
Maurice Eash 
Jim Sanders 

Gene Glass 

Ken Komoski 
Marjorie Edelson 
Gary Storn 
Barak Rosenshine 


(en) 


(.25) 


Accountability Notebook development 
Measuring educational priorities 


Effingham study 
Accountability Notebook development 


Other Contributors (Total = .75 FTE) 


Gini Gonsalves 
Doug Curry 
Ethel Mincie 
Marjorie Lee 
Val Summerville 
Michael Plog 
Mary Anne Bunda 


Digitized by the Internet Archive 
in.2022 with funding from oo Se 
University of Illinois Urbana-Champaign... 


https://archive.org/details/endofyearreportoOOstak 
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Table 2 


Proposed and Actual Activities 


Proposed Emphases 


In-service training 
Training materials and activities 


will be developed to aid gifted 
project personnel in the evaluation 

of their programs. Regional workshops, 
in-school meetings, and individual 
study will be considered as alternate 
approaches. 


Liaison 

CIRCE will increase its efforts to 
work together with OSPI officers and 
gifted project staff members across 
the state in matters of planning, 
demonstrating, and public relations 
as well as evaluation. 


Achievement Testing 
Within CIRCE a complete reorganization 


of the Illinois Statewide Testing 
Program is scheduled. It will be an 
objective of the Experimental Project 
to guide the efforts of these testing 
activities so that they are meaningful 
to the teacher--especially the gifted 
project teacher--who wants to monitor 
student progress in specific areas. 


Measuring Priorities 
The project staff will continue to 


study ways of measuring and reporting 
the priorities of a classroom teacher, 
a gifted project, or an entire school 
district. 


Actual Activities 


Training and Liaison 
Springfield workshop 


Champaign Ramada 

Jacksonville 

Classroom Report Follow-up 

Correspondence Course 
Try~out 


WwW & Ww hw Ee 


Development 
1. Development of Account- 


ability Notebook 

2. Distribution and Follow-up 
of Accountability Notebook 

3. Development of CMAS Films 

4. Evaluation Plan for 
Institute 


Basic Research 

1. Analysis of hazards in 
testing for performance 

2. Analysis of National 
Assessment 


1. Study of the differences 
between stated and opera- 
tionalized priorities 
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Table 3 


om el 


A Summary of the Allocation of Work and Funds 
for Each of the Main Objectives 
of the CIRCE Gifted Experimental Project, 1970-71 


Objective 
DEVELOPMENT OF ACCOUNTABILITY NOTEBOOK 
Intended O FTE 0 
Actual 1.00 FTE 16K 


DEVELOPMENT OF IN-SERVICE TRAINING MATERIALS 


Intended 1.25 FTE u Hy gi 4 
Actual -75 FTE a ly 
COMPARISON OF EVALUATION MODELS 
Intended ay eR ae 14 K 
Actual 0 FTE 0 
RESEARCH ON ACHIEVEMENT TESTING 
Intended sau. ELE 12 K 
Actual AF a ed gd 04 2°k 


DISSEMINATION AND FOLLOW-UP STUDIES 
Intended Ay eT VO 2k 
Actual -50 FTE my ae 


STUDY OF EDUCATIONAL PRIORITIES 
Intended -50 FTE 8 K 
Actual 00 ORLE 8 K 


MAINTAIN REFERENCE LIBRARY 
Intended -50 FTE 4K 
Actual -50 FTE 4K 

LIAISON WITH SCHOOLS AND STATE OFFI 
(including in-service training) 


Intended -50 FTE ony 

Actual » fo PLE 10 K 
ADMINIS TRATION 

Intended -25 FTE 10 K 

Actual -25 FTE 1k 


CE 


Results 


Analyses of techniques 
Self-evaluation plan 
DESDEG tryouts 
Lakeview plan 


Test and evaluation guide 


OSOT observations 


Experimental findings 
Instrument survey 
Working Paper #5 
Working Paper #6 


Forms 

Field test results 
Manual 

Teacher materials 


Working Paper #1 
Review of literature 
AERA paper (#7) 
Items for survey 


Library collection 
Files 
Newsletter 


Identification of needs 
Staff-briefing notes 
Working Paper #4 
Categorical aid paper 


Coordination 
Evaluation 
Final Report 


NOTES: FTE = Full-time equivalent for academic year; faculty and graduate 


students not distinguished.. 


All figures are estimates. 


year 1970-71 in thousands of dollars. 


Total FTE = 4.50. 


K = intended allocations for academic 


Total K = $72,000. 
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The Accountability Notebook 


The Accountability Notebook began with a dissatisfaction on the part 
of Robert E. Stake for the narrowness, rigidity, and negativism seen in the 
dominant view of education accountability. To counteract this unacceptable 
point of view, he designed a plan to produce an educational product which 
would enable school men to broaden, enliven, and make more positive their views 
of education accountability. 


To broaden their view, it was decided that the Accountability 
Notebook should be multidimensional, moving beyond the fiscal and legal respon- 
sibilities which school men recognize as central concepts in their view of 
education accountability. 


He sought to enliven the view of education accountability by making 
the product in a loose leaf notebook form, thereby somewhat encouraging the 
school man to grow in his view and to add or delete from the Notebook as he 
sees fit. 


The Accountability Notebook was to be a positive statement of the 
opportunities that were already at hand for the school man to demonstrate the 
ways in which he was being accountable to his public, to suggest standards and 
procedures for achieving a more accountable posture where necessary, and to 
encourage him to include local entries in his Accountability Notebook where 
they were already at hand. 


The plan called for four major sections (with three subdivisions in 
each) roughly entitled School Audit, Student Performance, Teacher Performance, 
and Administrative Performance. A preliminary conference to discuss the 
contents of the opening statement, the standards, the procedures, and the items 
to be included under each of these sections, was planned for early December. 

At that conference, elementary school, high school, and university people met 
to discuss the Accountability Notebook. Present were subject specialists, 
curriculum specialists, and learning, evaluation, and measurement specialists. 
Preliminary statements were drafted at that conference for three of the twelve 
sections, and the decision was made to secure the other nine statements from 
experts in those respective fields. From that point in December through to the 
first week in March when the first prototype of the Accountability Notebook was 
ready, the usual hundreds of man-hours were spent in securing the help of 
experts, writing the draft statements, seeking criticism from expert reviewers, 
rewriting, and in some instances rewriting as many as five times. 


During the month of March the Accountability Notebook draft was taken 
into three school districts for field testing: Urbana, Effingham, and Highland 
Park. The counsel we received from superintendents, business managers, 
teachers and others in the field was heeded in the final revision of the 
Accountability Notebook prior to its going to press. Some of the critical 
reviews which we obtained from informants in these three test sites were 
included in the opening statements of the Notebook itself. 


One thousand copies of the Accountability Notebook were printed and 
were ready for distribution by the first week in April. One hundred fifty 
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copies of the Accountability Notebook were delivered the day they came off the 
press to State Department Gifted Project supervisors. The following week all 
remaining gifted projects received an Accountability Notebook through the mail. 
In addition one hundred twelve copies were sent to the State Department for use 
with various service centers; they also received an additional hundred copies 
for their own use. During the past three months copies have been sent out for 
research and evaluation purposes to individuals, school districts, colleges, 
and corporations throughout the United States. At the time of this report 
CIRCE has on hand fewer than one hundred copies of the Accountability Notebook. 


In May of 1971 all Accountability Notebook owners received a packet 
of additional entries for updating their notebooks. These included a number of 
annotated references, information concerning performance contracting, and 
several checklists and summary charts (see appendix). Recipients were left to 
place the sheets in the Accountability Notebook sections where they most fitted 
their schools‘ particular needs and uses. 


And what of that final form? The organization of the Notebook is as 
follows: The four divisions (administration, teaching, students, school) have 
three sections each; each of the twelve sections has five subsections: a. A 
summary overview of the responsibility of the school man in a particular area; 
b. A statement of the standards that are available for assessing account- 
ability; c. A statement of the procedures which CIRCE recommends for use; 

d. Suggested entries to be made by the local school man in the notebook; 
e. A brief basic reference list, annotated for the user. The introduction to 
the Accountability Notebook discusses four basic dimensions to accountability 
as viewed by CIRCE: personal responsibility, attribution of programs for com- 
ponents to its products, evaluation, and public disclosure. In reality, dis- 
closure is probably a dimension included within each of the first three. 


CIRCE contends in the Accountability Notebook that once personal 
responsibility for program development or for a given task within the school 
system has been identified, then the accountable school man discloses the 
statement of responsibilities to interested parties. Attempts to find cause 
and effect relationships associated with the implementation of certain programs 
are disclosed as well, and the school man also discloses the results of evalu- 
ation studies he conducts to determine the merit and worth of his educational 
program. 


A full report of Accountability Notebock dissemination and follow-up 
procedures can be found elsewhere in this report (see Dissemination of Evalu- 
ation Materials section). 
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Dissemination of Evaluation Materials 


As it evolved, one of CIRCE's responsibilities for this past year has 
been the dissemination of evaluation materials to be used by Gifted Program 
personnel. Two instruments which were given attention this year are the 
Classroom Report, which had been developed by CIRCE for the Gifted Program last 
year, and the Class Activities Questionnaire, which was designed by Joe Steele 
three years ago. A third product, the Accountability Notebook, developed by 


CIRCE this year, was the subject of an extensive dissemination effort this 
spring. 


CLASSROOM REPORT 


Early last fall CIRCE conducted a telephone survey to a sample of the 
OSPI staff who had requested Classroom Report forms in late spring of 1970. We 
were distressed to discover that only a very few teachers were using the 
Classroom Report. Upon more extensive inquiry as to why the Classroom Report 
was not being utilized (questioning of many teachers and other staff personnel) 
we discovered that: 


1. Dissemination strategy utilized by state staff was poor. 
Many of the Reimbursement Directors simply had Classroom 
Reports (CR) dropped on their desks and were told to use 
them if they wished. 


2. The teachers who had seen it were not sure that it was 
appropriate to modify the CR. Therefore, if there was 
a section which disturbed them, they tended not to com 
plete the instrument at all. 


3. Many teachers were threatened by the fact that the CR 
was promoted as a reporting device. They were and 
still are not ready to commit themselves to the task 
of reporting on their classroom activities. 


4. One of the more often expressed comments was that com- 
pleting the CR was simply too time-consuming for the 
teacher, and that time could not be given to a new and 
unfamiliar instrument. 


5. On a positive note, severel taachors commented that 
completing the CR enabled them to logically think 
through and assess their classroom program. This 
self-assessment aspect of the Classroom Report had a 
certain appeal to them. 


Based on these facts, it became clear to us that if we wanted the CR 
to be used we needed to assume responsibility for promoting it (the original 
dissemination responsibility had been left to the state staff). A part of last 
year was spent in doing precisely that. The CIRCE staff used some of the 
year's workshops and training sessions to describe to state staff the use and 
function of the Classroom Report (see Liaison Activities section in this 
report). 
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Despite these presentations and personal consultations the Classroom 
Report was still not used extensively this year. Our experience demonstrated 
quite well that attempting to introduce teachers to new formal evaluation pro- 
cedures is no easy task. To address this problem, Robert Wolf has looked 
closely into that matter by engaging in a study of the role of the teacher in 
classroom evaluation. The purpose of this study is to discover how teachers 
feel about the role of evaluation as well as about the general notion of 
evaluation. Designed around the CR's low incidence of use, the study gathered 
judgments from teachers as to the practicality and worth of the CR and other 
more general evaluation issues. (This study will be made available to the OSPI 
state staff upon completion.) 


CLASS ACTIVITIES QUESTIONNAIRE 


The Class Activities Questionnaire (CAQ) was developed for use in 
evaluating the Illinois Gifted Program. Initial development and field testing 
occurred between October 1968 and March 1969, and data was collected in spring 
1969. A computer program developed at Northwestern was revised and adapted to 
the IBM 360 computer at the University of Illinois at Urbana-Champaign in the 
fall of 1969. 


In 1970 the instrument was requested for use in several hundred 
classrooms in Illinois. [It was supplied and processed by the evaluation 
project until November 1970. At that time the demands of the evaluation 
project precluded further servicing of CAQ requests. 


In 1970 and 1971 a number of researchers requested permission to 
utilize the Class Activities Questionnaire in their studies. Two researches 
were reported in papers at the 1971 Annual Meeting of the AERA. Studies are 
in progress in Connecticut, Georgia, Illinois, Massachusetts, Minnesota, and 
Newfoundland. 


Beginning in November 1970, discussions were initiated at four uni- 
versities in Illinois to arrange for agencies to supply and service CAQs on 
request. In June 1971 arrangements were finalized to establish two centers in 
Illinois to service the instrument. Dr. Robert Rosemier, Graham Hall, 
Northern Illinois University, DeKalb, Illinois, will direct the supply and 
processing of CAQs for the northern half of the state in cooperation with the 
Rockford Area Service and Demonstration Center, which will provide consultants 
and follow-up services. Mr. Richard Bodine, director of the Champaign Area 
Service and Demonstration Center will supervise the servicing of CAQs for the 
southern half of Illinois. 


Both centers will supply these services to the districts at cost. 
By contractual agreement with the Gifted Program Development Section of the 
OSPI, the state will provide funds to cover indirect costs until volume makes 
the operations self-supporting. The current estimate of the charge to schools 
for use of the instrument is $7.00 per classroom group of up to 30 students. 
Further information will be available from these centers in the near future. 
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Both centers have operational computer programs and are expected to 
be functioning in September 1971. The center at Northern Illinois University 
will supply all requests from out of state in addition to requests from 
Illinois teachers and researchers. 


The CAQ has been presented to groups of teachers on a number of 
occasions. (CEC 1970, AERA Annual Session 1971, California Association for the 
Gifted 1971, several teacher workshops around the country.) In the last year, 
training sessions in the use of the CAQ have been conducted with the regional 
consultants of the Illinois Gifted Program and with directors of all Area 
Service and Demonstration Centers in Illinois. None of these presentations 
produced many requests for use of the instrument. 


Why this apathy towards what appear to the evaluation staff to be a 
useful instrument for teachers? First, we hypothesize, there has been some 
doubt as to availability of services to score the instrument. It is difficult 
to score by hand. The mailing in of forms no doubt discourages use. A second 
probable deterrent is that no information is at hand as to how a teacher might 
use the information to change his behavior or develop activities more in line 
with his ideal. Similarly, the CAQ does not simplify any existing tasks of the 
teacher. It does not help evaluate individual students or eliminate other 
tasks the teacher must accomplish. If anything, it creates a threat to the 
teacher's self-esteem and requires considerable review and self-assessment, if 
the results are to be useful. 


A third element in its lack of use is that administrators are dis- 
couraged from using it to evaluate teachers. As its use directly benefits only 
the teacher himself and it does require both time and money, teachers may feel 
that their administrators would not support their use of it. Finally, internal 
difficulties and changes in the funding and organization of the Illinois Gifted 
Program also drew attention away from the utilization of this instrument. 


Where the instrument has been used, a state consultant or evaluation 
staff member has strongly promoted and encouraged its use. Such active promo- 
tion and advocacy seems to be required to get teachers to use it. Perhaps, a 
program of training should be developed to accompany its use in many situations. 
This training might well be built around the self-assessment approach and 
include the CAQ as one of a number of instruments to aid the teacher. It seems 
unlikely that the CAQ will enjoy much use without such supporting services and 
active promotion. 


ACCOUNTABILITY NOTEBOOK 


In dealing with Accountability Notebook dissemination, it was decided 
on the basis of past experiences that CIRCE would take a great deal of initia- 
tive in disseminating the Notebook. Dissemination strategy was developed with 
the intention of conducting a major follow-up project next year. CIRCE has 
used several means to disseminate the Notebook; next year its general usage 
will be monitored in part to learn whether certain modes of dissemination were 
more effective than others. The dissemination methods selected were as follows: 
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1. Received by Direct Mail - Approximately one hundred ninety 
Notebooks were mailed to Reimbursement Directors, Service and Demonstration 
Center Directors, and Area Service Center Directors throughout the state. 
These Notebooks arrived with a cover letter from the Director of the Program 
which made a reference to the Notebook but did not describe the possibilities 
of use. Staff personnel were left on their own initiative to decide ways the 
Notebook could be used to help them. 


2. Received Personally from State Staff Supervisor - Approximately 
one hundred sixty-five Reimbursement Directors in the Chicago area, and fifty- 


five Directors in the central part of the state, received their Accountability 
Notebooks directly from state staff personnel. It was hoped that this mode 
would engender a discussion between Directors and state staff as to potential, 
applicability, and the like, and that this discussion would encourage the 
Director to use and modify the Notebook to a greater extent. 


3. Received at Workshops on Accountability - Approximately eighty 
participants received their Notebooks when they attended one of the four 


Accountability Notebook Workshops held around the state this May. Attendance 
was not mandatory, but pressure has a way of being exerted, and voluntarily or 
not many Service and Demonstration Directors were there because they felt they 
had to be. The method of presentation of the Accountability Notebook differed 
somewhat at each workshop but the content was quite similar. The four basic 
themes of accountability (disclosure, responsibility, evaluation, and attri- 
bution) were discussed at length and asserted to fuse all dimensions of the 
Notebook. It was emphasized that the Notebook represented "one man's view" 
(CIRCE) and therefore might serve as only a structural guide to broaden the 
notion of accountability and should be modified to suit local needs; that as a 
looseleaf notebook it should be added to, or subtracted from. It was suggested 
that this Notebook be used by many people within a program; that it should not 
become a static document; that it should be shared not only with the people 
within the Gifted Program but also with superintendents, local school board 
members, and any other group with responsibility for school programs. There 
was not a great deal of involvement on the part of the participants--though 
most of them expressed enthusiasm over the Notebook itself. At any rate, 
contact was made with several directors who were extremely positive in their 
plans for utilization, and follow-up is planned for next year. 


These three modes of dissemination might prove to have a differential 
effect in terms of usage next year; this question will be interesting to 
pursue. It is thought to be important for us to determine the best means of 
disseminating evaluation tools for utilization by school people. This year we 
became much more sensitive to the need. Hopefully, next year we will become 
more adept with regard to that activity. 
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Liaison Activities 


During the 1970-71 academic year CIRCE's liaison efforts with the 
OSPI Gifted Program staff have been primarily concerned with the dissemination 
of certain evaluation materials. However, these activities have also included 
several other important tasks, such as individual workshops, site visitations, 
and extended consultation in specific local areas. 


Early in the fall several CIRCE staff members met with the state 
staff on three separate occasions to discuss activities anticipated for the 
coming year. At the onset it was mutually decided that the OSPI state staff 
would assume primary responsibility for training other project personnel with 
regard to utilization of evaluation products, and that CIRCE would provide the 
staff with the necessary support and counsel in such training procedures. The 
next two fall meetings were spent familiarizing the state staff with two evalu- 
ation instruments, the Classroom Report and the Class Activities Questionnaire, 
to facilitate their future training work. These initial meetings were signi- 
ficant in the sense that they shaped CIRCE's general liaison plans for the 
remainder of the year. Because CIRCE was to function in only a supportive role, 
the prime impetus for training was to come from the state. 


In December, responding to the state's suggestion, CIRCE housed a 
two-day workshop in Urbana for state staff members and directors of the Service 
and Demonstration Centers to update the attending personnel in certain evalu- 
ation concerns and issues. Because CIRCE anticipated several positive outcomes 
from such an activity, we agreed to orchestrate the workshop. One such antic- 
ipated outcome was that hopefully the CIRCE staff would have more personal 
contact with gifted program personnel so that several individual working rela- 
tionships might develop. It was also hoped that an increase in product utili- 
zation would occur if CIRCE played a significant role in the training 
procedures. During the workshop, the Classroom Report and the Class Activities 
Questionnaire were explained in depth, and strategies for evaluating evaluation 
proposals and reports were discussed. At the concluding session of that 
workshop, CIRCE staff members gave individual counsel to the various Service 
and Demonstration directors concerning their own project evaluations for rating 
evaluation plans. For these discussions CIRCE staff members used one~page 
check sheets entitled "Guidelines for Rating an Evaluation Plan." These rating 
sheets were later evaluated as to their helpfulness and utility (see appendix). 
The value of this individualized consultation aspect of the workshop is dubi- 
ous. There is a great need to closely examine the real feasibility, for both 
evaluator and consultee, of providing any kind of meaningful consultation 
experience without considerable understanding of the situation, the person- 
alities involved, or particular program needs. CIRCE evaluators noted after 
the experience that they had had only a cursory knowledge of the program, and 
felt that had they known more about the project history, its environment, and 
the relationships between project personnel and regional and state staffs, 
their efforts would have been more worthwhile. They sensed a need also for 
some kind of ongoing relationship with their consultees in order to provide 
some time in which to understand the project, its personalities, and its needs. 
One hour may have sufficed for the evaluators and their consultees to begin to 
probe general problem areas, evaluation ideas, and personal motives and commit- 
ments. But without further relationships through feedback or follow-up built 
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into the experience, the possibility of personal impact or real operational 
change was minimal. The factors of lack of follow-up and ignorance or socio- 
psychological dimensions seem to have greatly reduced our potential impact and 
forcefulness, possibly even to the extent of leaving negative impressions of a 
workshop consultation unevenly conducted. Some CIRCE specialists spent hours 
readying themselves for consultations that never materialized. Other consult- 
ants spent minutes and used their afternoon to chat of this and that. Poor 
business at best. 


The following is a candid comment made by one of the CIRCE evaluators 
concerning his perceptions after participating in the individual consultation. 


I am not sending you a checklist because it was not 
applicable to my situation. There was no evaluation "plan" 
in the proposal, only a bunch of gibberish that couldn't 
make the "weak'"' column of your checklist. 


None of us thought this was a worthwhile tool for 
this workshop. In fact, the director of District One 
had never even been able to obtain a copy of his proposal 
since he took over the position in September (I am sending 
him a copy). Upon reading the evaluation section in "his" 
proposal he realized how incompetent it was also. 


You are right...it was a good experience, but still 
symbolic of the hit~and~run-variety-show syndrome started 
by the state and seemingly continued by CIRCE. First a 
variety of speakore parades heofersa tho damanctration direc- 
tors with little if any thought given as to the short or 
long range results. (For example, you and I can derive 
some benefits from listening to Stake's National Assessment 
speech, but for these directors it was just like a course 
in Latin--good to be exposed but of no use in the real 
world.) 


I have not thought through the above comments; they 
are just an impression. However, it does seem to me that 
something is rotten in the State of CIRCE. Perhaps because 
I am not privy to the inner councils, I am unaware of 
future plans. But it seems that if any forethought had 
gone into this meeting, it would have occurred earlier in 
the school year and some provisions would have been made 
for follow-up by the CIRCE staff. Instead the workshop 
looks to me like a facade...an event staged to make people 
think that CIRCE has some inclination toward service or 
that CIRCE is earning the money it receives in its state 
contract with the Gifted Program. 


The next major liaison endeavor occurred in March when Robert Wolf 
addressed several large groups of Gifted Program personnel (ranging from 
classroom teachers, reimbursement staff, and Service and Demonstration 
directors to state staff supervisors) at the Annual Association for the Gifted 
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Conference. Again this effort was solicited by the Springfield office, and 
CIRCE complied with the request. The topic of discussion for these two 
sessions was teacher classroom evaluation, and again the Classroom Report 

was distributed and explained at length. As an outgrowth of those discussions, 
Wolf was asked to give individual consultation with local school personnel 
throughout the spring months. These local efforts included site visits to 
discuss evaluation plans for individual projects. 


The last major liaison efforts occurred when Terry Denny and 
Robert Wolf ran a series of four accountability workshops around the state 
during the month of May. The principal objective of these sessions was to 
discuss the applicability of the Accountability Notebook. Other related 
issues were covered. While these meetings were organized for the benefit of 
Reimbursement Directors, state supervisors, and area service center personnel 
were also present. 


One final but more subtle general liaison activity was the continuous 
dialogue which occurred throughout the entire year of this pxrwject tbotween 
CIRCE's project director, Robert E. Stake, and the director of the Gifted 
Program on a wide range of project-related issues. 


Before discussing more specific liaison efforts which CIRCE staff 
members have been engaged in this year, a question that needs to be asked con- 
cerning broad scale activities is whether these kinds of generalized efforts 
are most effective in dealing with public school personnel. School men may 
perceive this form of liaison as extremely condescending; they resent being 
talked down to by university-affiliated individuals from outside the public 
school domain. These perceptions may be ill-founded or quite valid, but their 
legitimacy is inconsequential, for it is a genuine feeling that clouds the per- 
ceptual screen and adversely influences behavior. 


Similarly worrisome is that large group sessions are often redundant 
and treat issues superficially, at best. Asking Gifted Program personnel to 
abandon local projects to attend such sessions may not be as rewarding as 
expected. We felt several times this year that workshop participants attended 
not because they desired to but because it was expected of them. We feel that 
this phenomenon can best be attacked by the state supervisory staff more than 
by CIRCE's efforts. CIRCE has been responsible for the content and structural 
aspects of the several workshops, while attendance has remained a function of 
the OSPI staff. However, CIRCE must begin to think of liaison activities as 
more than public relations or promotional efforts. Very rarely does public 
relations work evolve into meaningful dialogue or professional relationships. 
For that to occur both groups need to understand the potential benefits of a 
collective effort to institute change within a program. The developer must 
learn to appreciate the endeavors of the practitioner and vice versa. 


It is entirely realistic to suggest, based on several successes this 
past year, that the most fruitful liaison strategy might be in site visits and 
individual program consultation, coupled with extensive feedback. 


Some specific work in this area has been carried out this year in 
local areas around Illinois, and these efforts may serve to stimulate more of 
this activity in the future. Sites which received considerable individual 
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evaluation counsel from CIRCE this year were Decatur, Jacksonville, Spring- 
field, Urbana, and two sites in Chicago. The amount of help and counsel which 
CIRCE gave to these and other projects ranged from very little to a great deal 
of supportive help. One site merely acknowledged our repeated attempts to 
help (Piasa); another effort consisted of a series of three meetings which 
resulted in a mutual agreement to abandom efforts (Springfield); other sites 
blossomed into productive relationships of continuous help, including as many 
as fifteen site visits (Urbana and Decatur). A product of this sort of rela- 
tionship, established without mass meetings but through individual consulta- 
tion, may be seen in the three case studies written from the project at 
Illinois High School (see appendix). 


Efforts such as these liaison activities, especially with the facul- 
ties with which CIRCE established a strong relationship, will be continued next 
year. But even this seemingly more productive tactic of individual consulta- 
tion and site visits becomes ineffective as a state strategy unless the full 
description and evaluation process of those mutual relationships are made 
available through the state to other project sites. The need for some kind of 
information monitoring system is great, and should not fail to be explored. 
This prototypic liaison activity should also encourage better intra-program 
liaison and cooperation; hopefully the various projects may take the initiative 
to develop these relationships. 
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Jacksonville 


Glenn Bracht, Assistant Professor of Educational Psychology and 
Guidance at Southern Illinois University, presented an excellent possibility 
for in-service training for research assistants at CIRCE when he invited them 
to work with him in Jacksonville, Illinois, where he was to evaluate the 
school system's Independent Learning Program and their faculty training 
sessions. Two CIRCE assistants met with him in Jacksonville on February 19, 
1971 at which time he had planned to work with the program directors on speci- 
fically formulating his evaluation procedures. 


The assistants benefited from their experience in two major ways. 
First, active participation in the morning meeting with Bracht and the direc- 
tors gave them a well-rounded picture of the practical problems of communica- 
tion in the field. Second, they met with Bracht in the afternoon as he drew 
up some of the instruments for his evaluation that had been decided on in the 
morning conference. They were able to actively participate in forming workable 
instruments meant for a specific situation by working with Dr. Bracht on 
student and parent attitude questionnaires (see appendix). Close contact with 
a professional evaluation in a field setting working on specific problems of a 
site evaluation provided the research assistants with an excellent opportunity 
to test and round out theoretical knowledge and individual ideas. 
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CIRCE Resource Center 


The Gifted Project Reference Library and Resource Center began during 
the first year of the project and was continued and expanded throughout the 
second year. In October the collection was moved into a permanent room with 
adequate physical facilities, and work was begun to further organize and cross- 
reference the materials. 


An alphabetical card file was developed, categorizing all papers, 
writings, and reports by author and title if possible, otherwise by project or 
organization name. We began and are still developing a topical cross-reference 
system to facilitate wider use of the materials. At present, twenty-four 
subject topics, chosen for potential usefulness and logical grouping, are cur- 
rently cross-referenced (see appendix). 


Aside from these referenced materials (papers, unpublished writings, 
reports, proposals, projects, manuals) the Resource Center houses publications 
of CIRCE senior staff, a complete set of working papers, an education news- 
letter file, self-instruction series and model curriculum guides, and assorted 
books dealing with evaluation and measurement. 


It is hoped that, with continuing organization, and with CIRCE 
research assistants constantly housing and carding all incoming documents, the 
Resource Center will continue to serve as a central location and clearing 
house for not only CIRCE staff research and projects but to other persons in 
Illinois involved in gifted program evaluation. 
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Curriculum Analysis Training 


The problem of training teachers to do curriculum analysis was 
approached by bringing together individuals concerned and involved with cur- 
riculum analysis. The group convened in Boulder, Colorado, August 13-15, 
1970. Utilizing the curriculum analysis materials (CMAS) developed by 
Irving Morrisett and William Stevens of the Social Studies Educational Con- 
sortium of Boulder, Colorado, the conferees discussed curriculum analysis. 
The advantages of curriculum analysis, the problems of analysis, the possibil- 
ities available to teachers in the Illinois Gifted Program, and the “how to" 
of doing analysis were explored in the continuing discussions. All of the 
sessions were video-taped for the purpose of making a training film for the 
teachers in Illinois. Participants at the conference were: Robert E. Stake, 
Irving Morrisett, William Stevens, Gordon Hoke, Jim Sanders, Chris Buethe, 
Charles Caruson, Bill Hobson, Margaret Wilmot, Douglas Sjogren, and 
Terry Hopkins Elofson. 


The six hours of video-tape were cut and edited to a tape approxi- 
mately one hour in length which hopefully would concisely present curriculum 
analysis to viewers. Mr. Richard Mann did the professional editing. 


Two hundred fifty sets of materials were compiled to be used with 
the film. Included in the sets were three booklets from the Harvard Public 
Issues, S.S.E.C. Newsletters (explaining curriculum analysis) and outlines of 
curriculum analysis procedures. A package of instructions and worksheets was 
developed to coordinate the film and the sets of printed materials (see appen- 
dix). It is hoped that with these sets of packaged materials a teacher on her 
own or a group of teachers could view the film and try out the curriculum 
analysis methods. 
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Correspondence Course 


A trial correspondence course was begun by CIRCE early this fall 
with the intent of developing a workable method of dealing directly with 
classroom teachers in problems of evaluation. Terry Hopkins Elofson compiled 
a starter set of basic evaluation materials which, it was hoped, would help 
teachers understand evaluation theory, develop basic evaluation skills, and in 
general become more comfortable with the concept of educational evaluation. 


The first Study Kit included the CIRCE Attitude Scale and Self- 
Scoring Sheet, a chart of Prototypes of Evaluation, a short evaluation bibli- 
ography, and three readings: '"Generalizability of Program Evaluation" and 
“The Countenance of Educational Evaluation" by Robert E. Stake, and "The 
Methodology of Evaluation" by Michael Scriven. 


The Kit was sent to four women in Charleston, Illinois, an SD direc- 
tor, a reading consultant, an elementary school principal, and an early 
childhood program teacher. They read the materials and later discussed them 
with Terry Hopkins Elofson. In summary, the reactions to the materials were: 


CIRCE Attitude Scale: frustrating to answer; changed minds frequently; 
provided good orientation to evaluation. 


Readings: found Stake's "Generalizability" paper easiest; 
Scriven's was most difficult; readings shovid have 
a Ladies Home Journal approach; felt need cf more 
confidence in evaluation; insecure about statistics 
and analysis. 


Prototypes: interesting; liked the simplicity.of chart and 
easily understandable category breakdown. 


Other than the first Study Kit and first discussion session, little 
more was accomplished with the Charleston women. The original intent of the 
course--a self-study and self-help program with exchange of ideas to CIRCE and 
back in problem areas--was never realized. The general neglect of both the 
teachers and the CIRCE staff in general may indicate a need for better trans- 
latability from the researcher to the practitioner, and a furthering of aware- 
ness of the more specific problem areas of the classroom teacher. 
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Evaluation Design of a Special Program for Talented Youth 


In conjunction with the Twin City Institute for Talented Youth 


(Minneapolis-St. Paul; Charles Caruson, Director), the following data~gathering 
design was developed: 


Pe 


fe 


7. 


8. 


10. 


11. 


12s 


Description of the Setting. Emphasis on the physical facilities, building 
and grounds, field-activity sites, perhaps with the inclusion of a map. 
Location of administrative offices, teacher offices, supplementary teaching 
resources, storage, etc. 


Instructional Transactions. Summary of what happened, what sorts of expe- 
riences were planned and carried out, learning environment, field trips, 
etc. 


Rationale. The philosophy of the special program. Basic expectation of 
needs of young people; ways of fostering individual growth, talents, and 
limitations of teachers; role of other students in student learning; etc. 


Hopes, Expectations, Fears. Summary of the intended inputs, transactions, 
outcomes. A more detailed statement of item 2 with reflection of the 
statements of item 3. 


The Calendar. A record of what happened when. A schedule of a typical 
day. 


Class Aims, Accomplishments. A class-by-class descriptive listing of what 
goals are pursued and what techniques are used. Master teacher, associate 
teachers are identified. Comments by the evaluation team as to level of 
success attained. 


Emphasis on Cognitive Talents. A description of the various classrooms as 
indicated by the Classroom Attitude Questionnaire. Indications of level of 
cognitive discourse, amount of teacher talk, etc. 


Emphasis on Creative Talents. A descriptive of the opportunities and con- 
straints on creative expression by individual students and student groups. 


A Student's View of the Program. One student's essay of the short-answer 
final critique completed by all students. 


The Students’ Collective View of the Program. A summary of the short- 
answer final critique completed by all students. 


A Participant Observer's View of the Program. The report of a student 
charged with the responsibility at the outset of the program of keeping a 


diary or notes on events and impressions. 


The Teachers’ Collective View of the Program. A summary of the final 
pencil-and-paper debriefing of the staff. 
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13. 


14. 


15. 


he 


Special Reports on Unique Aspects of the Program. Such items as the 
admissions policy, the management, the open campus, the field trips, the 
in-service training component, and the influence of this program on other 
community programs. Done where possible by specialists brought in just 
for this purpose. 


Adversary Statements. A summary of all the arguments supporting this par- 
ticular program and a summary of all the arguments contesting this partic- 
ular program. Left without resolution for the reader to weigh and 
interpret. 


The Evaluation Plan. A statement of the purposes, constraints, personnel 


and other resources, and plan of operation of the evaluation team. 
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The Comparison of Evaluation Models 


The proposal to do work in 1970-71 carried with it the intent to 
conduct a comparative study of the attractiveness, utility, costs, and bene- 
fits of employing at least two and possibly more evaluation models in a few 
Gifted Project sites. The project never progressed beyond the stages of model 
and site selection. 


The main reason for aborting the project lay with a mismatch of CIRCE 
talent with proposed work requirements. The Gifted Projects who had used 
DESDEG in 1969-70 did not wish to continue its use; and the junior staff 
trained in evaluation counseling skills (Dennis Gooler, Theresa Hopkins 
Elofson) were utilized elsewhere on the project. However, the principal CIRCE 
senior staff who was to be involved, Douglas Sjogren, left the University, and 
other staff members were unwilling and unable to complete the project. 
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Achievement Testing 


The proposal for second year (1970-71) included the following item 
indicating one of four principal areas in which CIRCE was to be involved: 


3. Achievement testing. Within CIRCE a complete 
reorganization of the Illinois Statewide Testing 
Program is scheduled. It will be an objective 
of the Experimental Project to guide the efforts 
of these testing activities so that they are 
meaningful to the teacher--especially the gifted 
project teacher--who wants to monitor student 
progress in specific areas. 


This objective was not pursued. The Evaluation Specialist (Budget 
Item No. I-A-3) sought by CIRCE and the Gifted Project did not choose to accept 
the position offered. Without this person the reorganization of the Statewide 
Testing Program moved only a small step forward and the objective for the 
Gifted Experimental Project was switched to the development of the Account- 
ability Notebook and of a statement of cautions appropriate for achievement 
testing in performance contracting situations. A paper (see appendix) 
entitled "Measuring What Students Learn" points out the difficulties of using 
gain scores as indicators, using grade equivalents as units, and so on. 


Some time from this study was also devoted to a study entitled 
"National Assessment" (see appendix). These two papers, both written by 
Dr. Robert E. Stake, were distributed across the state to Gifted Project 
directors. 
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Curriculum Evaluation 


Edward Kelly confronted the problem of uniqueness and transfer of evaluation 
methods and presented his writing as a doctoral dissertation. 


ABSTRACT 


Curriculum Evaluation and Literary Criticism: The Explication of an Analogy 


Beginning with the assumption that curriculum evaluators can profit 
by watching how evaluations are carried on in other fields, the paper discusses 
the ways curriculum evaluation is similar to literary criticism. The discus- 
sion focuses on three points. They are: (1) the role that description plays 
in criticism and evaluation; (2) the role that judgment and justification play 
in the evaluative process; and (3) the importance of language in both criticism 
and evaluation. 


The consideration of description warns against the over-simplifica- 
tion of things like the classroom and its people, and suggests that there are 
three broad categories of descriptive data that may be used to reconstruct an 
object. The question of how one can know whether or not an object has been 
successfully described is discussed in detail, and some recommendation offered. 


Troubled by the implicit occurrence of criteria that frequently 
underlie judgments of worth, the consideration of judgment presents a process 
for the development of publically reasonable expressions of judgment that 
should help all parties to the curricular dialogue come closer to understanding 
why certain judgments have been made. 


Although a concern with the language of curriculum evaluation marks 
most of the third consideration, the principal focus is on the development of a 
rhetoric of the curriculum that shows how six concepts from the realm of 
literary criticism (metaphor, point of view, plot, action, structure, theme) 
can be used to critique a curriculum. Developing a consideration of the cur- 
riculum as one that has much to do with "experience," the consideration of lan- 
guage builds a rationale for talking about curriculum in ways that may help 
more people understand what is being said. The paper concludes with the 
warning that a move toward a criticism of curriculum should be seen as an 
adjunct to the presentmethodology of the curriculum evaluators, and not as a 
replacement. 
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Measuring Priorities 


Work concerning education priorities resulted in a thesis by Dennis Gooler. 
Two hundred copies of the thesis were distributed to Reimbursement and Service 
Center personnel who requested then. 


ABSTRACT 


Strategies for Obtaining Clarification of Priorities in Education 


Problem. One of the most pressing issues in education at the present 
time is accountability. Schools are coming under increasing pressure to be able 
to provide evidence of what they are doing, and how well they are doing it. 


The present accountability emphasis appears to be on assessing the 
outputs, not the inputs, of education. However, accountability must consider 
not only the extent to which goals are achieved, but the relative importance, 
or priority, of those goals. Therefore, the identification or clarification of 
the priorities assigned to potential goals of education, by various kinds of 
people, represents a significant task. 


This study is a report of an attempt to determine how to get good 
answers to the question: What do various kinds of people regard as the most 
important business of the school? To do so requires an understanding of the 
complexity of educational priorities, including how priorities are formulated, 
what influences those priorities, and how priorities might be measured. 


Critique of Related Studies. Several studies of educational prior- 
ities have been conducted, the most extensive being Downey's Task of Public 
Education. That study asked a variety of people to rank, by using a Q-Sort, 
sixteen broad educational goals. No precedents were found for school observa- 
tion as a means of determining priorities, nor were studies found that 
attempted to determine priorities at different levels of goal specificity. 
Maguire suggested that priorities might be stated in a number of different 
ways. That suggestion was explored in the present study. 


Specific Questions Investigated. The investigation reported in this 


study derived from the broad question of how we can get good answers to the 
question of what people think the goals of the school should be. What is 
important? To whom? Two broad areas of concern guided the study: 


1. What are some aspects of priorities that should be taken into 
account in a study of what people think the school ought to be? 
Basically, what are priorities? 


2. How can we talk about priorities? What language can we use? In 
addition, attention focused on ways to actually assess priori- 
ties, and ways of representing or displaying what was found. 


The study was an attempt to find out how to examine priorities, not 
on what priorities were actually held by various people. (Some initial data 
about actual priorities is reported in appendices.) 
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It is felt that people do have priorities, that those priorities seem 
to influence the way people behave, and that not a great deal is known about 
how those priorities can be measured. 


Two approaches to the study of educational priorities were used in 
the study. The author spent a number of months observing in a school to learn 
something about what ought to be considered in an attempt to measure priori-~ 
ties. A survey was also conducted in an attempt to gain some substantive 
knowledge about what people think the goals of the school should be. 


It was decided to "live" in a school system, to carefully observe 
and record what happened in a school, with the expectation that some insight 
concerning goals, priorities, and indicators of preference might be obtained 
that would verify or supplement original thinking about priorities. The ulti- 
mate intent was not te describe the classroom minute-by-minute, but rather to 
observe and record information about aspects and indicators of priorities, and 
to reflect on measurement problems that might be important. 


The survey was designed to examine the results of asking people to 
indicate their priorities in different ways, and to explore differences in 
priorities held by different kinds of people. 


Figure 1 shows the design of the survey. The figure indicates that 
people were asked to respond to three levels of goal specificity, and were 
asked to indicate priorities in three ways: importance, time allocation, and 
money allocation. In addition, people were asked about how things are (REAL) 
and how things ought to be (IDEAL). Instruments were designed to collect data 
in each cell. Rank order of different kinds of goals were determined, and 
mean rating profiles were developed. 


Conduct of the Investigation. The intent of observing classrooms in 
this study did not seem to fit any of the classroom observation procedures 
received. Thus, it was decided to simply report what was happening in the 
school, to record activities, comments, and actions of teachers and students. 
The researcher wanted to move with as little predetermined structure as possi- 
ble. He was looking for some indicators of goals, of preferences, of redun- 
dancies, but was unsure of the form those indications might take. The actual 
recording of classroom events was done on blank paper, divided to use half to 
report what happened, and half to note the researcher's reaction at the time. 
Teachers and students were often asked to clarify something. Each day, the 
notes were analyzed, and a summary made. In addition, the researcher actively 
intervened with questions at several points near the end of the study. 


Survey instruments were developed, tested, revised, and then adminis- 
tered to two groups of teachers, and to a number of community members. Recom- 
mendations for changes and additions to those instruments are suggested. An 
extensive analysis of the data obtained was made. 


The results of both aspects of the study are related to each other, 
together with recommendations for continued study in the area. 
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Analysis of the Data. Goals pursued in the school can be categorized 
into four broad areas: content/methodology, outcomes, processes, and adminis- 
tration. Priorities appear to be shaped by a number of constraints of forces: 
personal values, knowledge of alternatives, time, money, and pressures of regu- 
lations from outside agencies. In addition, social problems appear to influ- 
ence priorities. Priorities may change when different constraints on 
educational programs are present. 


There are several ways of talking about priorities (resource alloca- 
tion, time allocation, etc.). Priorities, as stated in these different 
languages, appear to vary. 


There is little change in Real-Ideal priorities. Most people indi- 
cate that more importance should be given to most goals than is presently true, 
but that the relative importance of goals would change little. Respondents in 
this study ranked goals almost identically to the ranking of goals in Downey's 
1960 study. 


It was found that a group of teachers, given the same goals in simu- 
lated settings quite diverse, tended to assign the same priority to those goals 
regardless of the situation. 


Much more extensive analysis of the survey data is included in the 
dissertation. 


Findings. The following conclusions were made as a result of the 
study: 


1. A study of priorities must include goals from each of the 
four broad areas (content/methodology, etc.). 


2. The most adequate representation of priorities may be a com- 
posit of each of the ways of talking about priorities. 


3. Priorities are dynamic, changing in light of social problems, 
new knowledge, changing value systems, new regulations, etc. 
These forces must be considered if priorities are to be 
meaningfully assessed. 


4. There does not now appear to be a single indicator of 
priorities. 


5. An understanding of priorities appears central to explana- 
tion of what a school does. 


Implications of the Results. Perhaps the most important implication 
of the research results is that accountability is a multifaceted concept. Dif- 
ferent people have different priorities, and these priorities appear to be 
subject to change. Thus, that which the school will be held accountable for 
may undergo constant change. 
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A second implication is that priorities may be manipulated. It may 
be possible for the school to gain support for a program by admitting new 
knowledge to its constituency, knowledge that may alter the percaptions of 
relative importance of goals. 


Finally, it appears that our methodology for collecting judgmental 
data is still rather primitive. Such data may be extremely important for eval- 
uation studies. 


Significance. The significance of the study is its contribution to 
the realization that goals and priorities, intricately a part of the process 
of educating, are complex phenomena that cannot and should not be reduced to a 
simplistic perception of reality. The school's task will be misrepresented if 
defined only in terms of a few desired cognitive learner outcomes. 


The study also contributes baseline data needed for further research 
into what people think is important, and how these feelings can be used as 
input to planning educational programs. 


New insight has been gained into those factors which must be con- 
sidered in our attempts to understand what people regard as the important 
business of the schools, and how those perceptions are developed. 
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Report of the Evaluation Committee: 
CIRCE Experimental Gifted Project 


The evaluation team met with CIRCE staff on June 30 to review their 
activities with the gifted program during the past year. The report includes 
our. reactions to each of the project's five major activities and a final 
section of general comments. 


Product Development Follow-Up 


The Classroom Report was developed during the first year of the 
project and then disseminated to teachers and administrators in gifted 
programs. This year a follow-up was begun to obtain information about who is 
using the Classroom Report and explanations for its use and non-use. The 
follow-up study will also give some indication of how teachers and adminis- 
trators generally feel about evaluation and available evaluation tools. 
Although the results of the follow-up are not yet available, recommendations 
should be forthcoming about procedures for effective dissemination of evalua- 
tion techniques and instruments. 


Accountability Notebook 


Although the Accountability Notebook has been developed for use in 
the state gifted program, it will have application for teachers and adminis- 
trators in all areas of education. The Notebook will stimulate people to 
think through what it means to be accountable. This can be done in group 
sessions as well as individually. Hopefully, it will also stimulate action in 
schools to disclose more information which displays their accounting activity. 


It is difficult to predict who will use the Notebook and how it will 
be used. Educators are not expected to use all sections of the Notebook to 
the same degree. Some will place greater emphasis on fiscal accountability; 
others will place greater emphasis on moral accountability. A strength of the 
Notebook is that it can be adapted to each individual's use; its organization 
makes it convenient for adding pages to any section. 


We expect the Accountability Notebook to broaden the user's views of 
accountability and become a primary resource and reference notebook for hin. 
Since use of the Notebook will depend on its dissemination, we are pleased 
that CIRCE staff are studying alternative methods of distribution and orien- 
tation to the Notebook. 


Liaison and In-Service Training 


The liaison and in-service activities included (1) informal contacts 
with teachers and directors of gifted programs throughout the state, (2) in- 
service workshops for state personnel and directors of service and demonstra-~ 
tion centers to orient them to the Classroom Report, the Class Activity Ques- 
tionnaire, and evaluation techniques, and (3) the Accountability Notebook 
institutes. 
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Although scattered informal feedback indicates that the in-service 
workshops were perceived as useful, the workshop leaders could have obtained 
judgment data from the participants. A formal evaluation could also have 
been conducted for the Accountability Notebook institutes. The following out- 
comes appear to have resulted from the liaison and workshop activities: 


1. CIRCE has obtained information about the problems that teachers 
and administrators have in evaluating gifted programs, the use- 
fulness of the Classroom Report and Accountability Notebook, 
and the attitude toward evaluation. 


2. The state staff in the gifted program are more capable of 
disseminating evaluation instruments to teachers and project 
directors and are providing more effective liaison between 
CIRCE and the gifted program. 


In addition, the direct contact between the CIRCE staff and the 
teachers and administrators of gifted programs should improve the credibility 
of evaluation and CIRCE. 


CMAS Videotape and Film 


Since teachers and administrators are constantly faced with the need 
to select educational materials, a project was undertaken to provide training 
for teachers in making an analysis of curriculum materials. The film which is 
being produced to explain the Curriculum Materials Analysis System (CMAS) 
should provide important training for teachers and administrators who select 
instructional materials. 


The videotape should be tried with several groups of teachers whose 
specialties cover different subject matter areas before the presentation is 
put on film. Although the videotape presentation has been effective in 
graduate courses, we feel it should also be tested with teachers in their 
local school settings. 


Measuring Priorities 


It has become apparent in recent years that not all segments of our 
society value the various goals of education to the same degree. The purpose 
of this activity was to develop instruments which are useful for measuring 
the educational priorities of different groups. 


We feel that the instruments developed for this project are valuable 
and should receive wider dissemination. Since the original report is probably 
too bulky for dissemination, we recommend the preparation of a manual to accom- 
pany the instruments. Although a priority study has been conducted in several 
Illinois communities, other communities should be encouraged to carry on their 
own studies rather than generalize from the communities which have already been 
studied. 
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General Comments 


The major activities of the Experimental Gifted Project during the 
second year have substantial value for the evaluation of gifted programs. The 
value of the product development follow-up, liaison, and in-service training 
activities is already evident. The Accountability Notebook, CMAS film, and 
priority measures have extensive potential, but their usefulness in the 
gifted program has not been demonstrated yet. 


During the two years of this project, CIRCE has developed several 
evaluation instruments for gifted programs. Since these instruments have 
little value unless used effectively, we are pleased with the emphasis placed 
on product development follow-up and in-service training during the past year. 
We encourage the state staff to place high priority during the following year 
on dissemination of the evaluation instruments which have been developed by 
CIRCE. 


The relationship which was developed between CIRCE and the state 
staff during the past year should contribute to the evaluation of gifted 
programs. A beginning has been made in giving state staff the confidence to 
assume the responsibility for evaluation of gifted programs. However, addi- 
tional training should be provided during the next year. 
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Guide for Rating an Evaluation Plan 


On the second day of the December 1970 CIRCE evaluation workshop, 
individual conferences were planned so that Service and Demonstration Center 
directors might discuss with CIRCE members the evaluation sections of their 
SD center proposals. Thirteen CIRCE evaluation specialists teamed with the 
twenty SD directors who participated; for each SD proposal to be discussed, 
CIRCE staff members were instructed to use, as informal rating sheets, the 
"Guide for Rating an Evaluation Plan." (see below) 


Of the twenty proposals that were individually discussed, ten CIRCE 
rating sheets were used. One of those was left with the SD director as a 
guide; the other nine were returned, many with comments concerning the useful- 
ness of the checklists. Most of the CIRCE evaluators’ comments implied that 
the checklists themselves were inappropriate; others leaned toward the proposal 
itself as reason for the inapplicability of the checklists. For whatever 
reason, fully half the evaluator-director teams found that the checklists were 
not usable as guides for rating the evaluation sections of the SD proposals. 
Some CIRCE comments: 


Checklist did not apply as there were no evaluation plans. 
All ratings were either "weak" or unknown. (Steele) 


Suggestion that these criteria are too "stringent" for 
application to an evaluation "subsection" of a proposal 
for refunding. (Kelly) 


I gave the checklist to the client, indicating what a 
proposal should encompass. Since their evaluation is 
almost a research project, it isn't fair to use the 
checklist on their plan. (Bunda) 


The checklist...was not very useful. The evaluation sections 
of the project proposals did not in any way match with the 
questions being posed by the project directors, or the inten- 
tions of those project directors. I began to use the check- 
list with the first group, and it did not seem to make any 
sense to them...they indicated that “well, you know, it 

was a proposal and we had to write something." Both project 
directors said that they did not know really what was in the 
proposal. It had been written a long time ago or it had been 
written by someone else. (Gooler) 


Other CIRCE members simply commented that the checklists were "not relevant" 
or "inappropriate." 


The rating sheets, however, were in the hands of CIRCE staff, and 
seem to have been at least looked at during the individual conferences. They 
did reportedly have some positive uses even though they might not have served 
as valid or appropriate instruments for rating the SD proposals. They were 
helpful as both stimulators of discussion and as informal guides of sorts for 
participating SD directors, and served as vehicles through which SD directors 
were forced to at least look at the otherwise unfamiliar evaluation sections 
of their proposals. And that CIRCE was able to categorize and itemize its cri- 
teria for rating evaluation plans served as tangible evidence that CIRCE 
members did have some definite guideposts for commenting on the SD proposals. 
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Our rating sheets, however useful in determining strengths and weak- 
nesses of more detailed and comprehensive evaluation plans, seem to have been 
far too discriminating for use in rating the often sketchy evaluation sections 
of SD proposals. 


We received a great deal of information that would seem to indicate 
that the proposals we were evaluating were simply inadequate. Many of them 
were too short or general to rate; others received a great many "weak" ratings. 
From the professional evaluators' standpoints, they seem to have been, indeed, 
disappointing. This may indicate the inadequate of the proposals; it may also 
indicate the great difference in the emphasis placed on the evaluation sections 
of those proposals. What was the object of our major thrust of efforts and 
talents was apparently, to SD directors, only a few paragraphs of writing, 
relatively unimportant within the refunding proposals. Conditions for refund- 
ing stipulate inclusion of an evaluation subsection: that the proposals con- 
tained evaluation sections was no guarantee of an interest in or a commitment 
to good, honest evaluation. We based an afternoon of individual discussion on 
the assumption that evaluation was important to them and to the effectiveness 
of their proposals. This appears to have been a questionable assumption. 


What was the intended use of the CIRCE checklists? If our intention 
was to give SD directors more awareness of and interest in their evaluation 
plans, we did, at least for the duration of the participating SD directors' 
individual conferences. We have no evidence or carryover or transfer. If our 
intention was to teach SD directors how to write an evaluation section for a 
proposal, the checklists to some extent did that, in that they provided an 
outline for the contents and major points of an evaluation. If our intention 
was to point out strengths and weaknesses of the proposal for the purpose of 
eventual alteration of the evaluation sections, we needed to have built into 
either the meetings or the instrument some means of successful follow-up. If 
our intention was to effectively evaluate the checklists we used in rating the 
proposals, we would have received better feedback concerning the checklists 
had both CIRCE evaluators and SD directors been informed of this intention. 


The rating sheet seems to have been a useful tool to stimulate dis- 
cussion and as information topic guide to promote exchange of ideas within an 
individual discussion. Their use for evaluating the proposal or for alterating 
and improving the project is not recommended. 
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Communication: 


Purpose: 


Technique: 


Practicality: 


CIRCE Evaluators 


Kelly 
Kerins 
Denny 
House 
Stake 
Steele 
Bunda 
Collier 
Walstrom 
Hopkins 
Wolf 
Gooler 
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Checklist Results 


Strong Satisfactory 
20% 50% 
18% 36% 
13% 37% 
13% 58% 


Project Directors (SD) 


Stark, Moore 
Mishken 

Hutchins, Buckholz 
Rockel, Postl 
Powe, Todd 
Horsbruck, Peters 
Bodine, Brinkhoff 
Habeeker 

Gotch, Laughlin 
Jeter, Farrell 
Fry 

Varby 


GUIDE FOR RATING AN EVALUATION PLAN 


COMMUNICATION 
Clarity: Does the plan read well? explicit? free of jargon? .........e.e.eee4 
Integrity: Do its pieces fit together? sense of unity?. .......+e.sees8 2s eee 
Seductivity: Does it persuade the reader to take part? PULL CUIN ST ou wie sls fe os 
PURPOSE 
Rationale: Does it answer "Why evaluate?" are goals explicit? ..........4s.+4.. 
Plan: Are activities outlines? are plan and rationale consistent? ........... 


Complementarity: Is it compatible with the program? with other studies? ........ 


TECHNIQUE 


_ 


Variable selection: Are selections relevant, imaginative, flexible?. ..., 


me 2 


Data gathering: Are efforts design-oriented, valid, redundant?. ......... 


e 
e 
° 


Reporting out: Are clients, audiences specified? responsibility?. ..... 


PRACTICALITY 


Scope: Is the plan feasible? are staff, facilities available? .......... 2 «ele 


Cost: Are estimates reasonable? are benefits worth cost?. ...... foacet 


eee See ee ee ee ee 
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Balance: Are resources nicely allocated? spread too thin?......... 
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Curriculum Materials Analysis System 


In-service Training 


MATERIALS TO BE USED WITH THE FILM OR VIDEOTAPE PROGRAM 


Produced by Robert E. Stake 
Center for Instructional Research and Curriculum Evaluation 
Gifted Experimental Project E-116 
University of Illinois 


Spring, 1971 


Terry Hopkins, General Editor 


State of Illinois 


Michael Bakalis 
Superintendent of Public Instruction 


Draft Prototype to Accompany Videotape 
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Please read this page before viewing the program. 


These materials are to be used with the film Curriculum Materials 


Analysis System. Don't use them unless you have the film. The materials and 


film can be studied by one person or by a group of people. 


The film has a discussion of the procedures and uses of the CMAS; the 
SSEC Newsletter also describes the CMAS; the Harvard Public Issues booklets are 
to be analyzed as a practice exercise. The directions for using the Newsletter 
and Harvard Public Issues booklets with the film are given in the film and in 


this packet of materials. 


Time required: Set aside two hours for working through the program and mate- 
rials. When the projector is ready to go, here's what you should do: 
1. Watch and listen to the program while CMAS is introduced 
and described. It won't be necessary to take notes or 
use these materials while the program is underway. 
2. After about 10 minutes, you will be asked to interrupt 
the program, turn off the projector, and return again 


to this set of materials. 


Now turn on the projector. 
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ONE 


(15 minutes) 


You have heard and seen a description of the main headings of the 
Curriculum Materials Analysis System. Now you should turn to the Newsletter 
from the SSEC (Social Science Education Consortium). Please spend about ten 


minutes readying pages 1-4 of the Newsletter. 


Then take five more minutes to think about the questions below. Make 


brief notes of your own tentative answers to the questions. 


1. Who benefits from curriculum analysis? 


2. Do you need a system like this? Why? Why not? 


3. Curriculum materials are not the whole of a curriculum. 
Is this system useful only with the materials? If not, 
how else is it useful? 


Have fifteen minutes passed? Turn on the projector. 
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STOP 
TWO 


(20 minutes) 


How might you answer the question: 


Whose opinions, whose judgments should be considered? 


Spend less than five minutes on this part. List the names of as many 
people you can think of whom you would trust to make judgments about curriculum 


materials for your class. Start the list with your own name. 
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Do you feel that these people would agree pretty much on what the 


curriculum should be? 


Turn to next page. 
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Now try your hand at analyzing some actual curriculum materials. You 


will need to find one booklet among these materials: 


Got it? Turn to next page. 


, Af 
abet Zou ask ao aha eum oe 


n me 
y iB A v : 


{ i : ’ a | 5 He ,, iY ; . ee 
y 4 ly . Je J ne hi . i tod i ihre We aan ie mit # vied " ici) 

‘ te iy ty i y) ( ow gf pe Mu F i % 1 pr A‘ ae ih ta} " ZT ni a. » nf ( M , ma 
| ; | | L ) F ‘ ' ae / | { | * ? ith a Aue, hl 
Mav he, OAR Sapa iene NA ih sili Ne ih ty Hi.) p ana ty hn 


if ; ae 
sina he } ‘i cil ity hed ” 
Ne iy ; : a 7 
yeaa ins : ’ levee f 
Fi eam y ) Aor 
p mA . bide, ns Fi i 
) bi i 1 i i iy . ‘ 


a. . Avira vee oli sets eh Oy MAb de va * byw ny Dy Ser entaet ‘i ih 
; Re All 


4 Pe oe UP 


We, 
ee ree i ere ath ibaa, mak wodlideiaee wets 4 
-t f vit i - ° opin 4 ; ‘ . 
le rie a : 
| ian \ 1 A Vane 
Be Dr NP Me triad > a ti Aik enn AY He ss bi bogie cae mate aoe 9 tio 
ate rien an : 8) i Vay heer 
fee lel Anti tal eheny hy Hear Oe ron bieeerappt yh mittee 1 
‘ ; } OAs i) 7 . 
; ayus 5 whi i Ni 
ia 
ee en ne Bias Viper 4 ae aresk arek Lele eet 
} , y \ a D 
Me ; dient yr a i oy iy iF 
: vt i's PAY 
H ne Thm Ag nels whiney 7 aired al “a or ri err rel reac Y 1 7 
v4 , i é F Ty, 7 : 
(Gu f i, ti ao ne if , 
a , y 
Puy t é ‘ ‘ 
TT ee | arene Seegiaherttghy 4) 'y mt . Lah vf 
Col 9 al f 5 : ‘ 
Ay ' 
| PS ees BY Pyl | 
ty ; pene A f Wee Nile dike open sang ifs sy Mey wil" Ve aM 
tt ‘ ; } ! ; I 
, bx ie eee Li ‘ i 
; Ao eee A ip | 4) WAR ay ti 
t : at j ' +i) pays a7 
J ' is - : ny ’ 
tte adele geiy en eer aN, Abeta m sa 
i , , i 
nA) } A ae (ae me) My ie x 
MMC oh 
; Ltt ‘ie 
ih he be a 
' a 
Py u F 
i 
1 y 4 At) 
On : oe. J ¥ a. A 
jj } i Mt 
; ; ft di, oir 


3) AE alias ee he ie a sn opi #i | iw 
. / K ‘ ea HiAy ™ Al 
We te tg A, tr lana 


y a OF XC ; : pu Tw Be my) Nee 1 ieee en 


i ie 


a Su 


Read page three of Cases and Controversy. As you read, think about 
the CMAS topic "2.0 Rationale and Objectives.'"' Jot down your answers to the 
following: 


1. The focus of the Public Issues Series/Harvard Social 


Studies Project is: 


2. What are some anticipated outcomes for students who 
use these materials? 


These answers of yours would be entered in Section 2.0 1f you were 


doing a CMAS analysis of Harvard Project Physics. 


Twenty minutes up? Finish watching the program before turning to the next page. 
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After viewing the film. 


Two units from the Public Issues Series/Harvard Social Studies 
Project are included with your materials. After scanning Taking a Stand and 
The American Revolution, write a brief statement for each about the CMAS topic 


"2.0 Rationale and Objectives." Do these booklets exemplify the rationale and 


objectives of the Public Issues Series/Harvard Social Studies Project? 


Taking a Stand The American Revolution 


Rationale 
and 


Objectives 
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We want to know if you found the experience a useful one: Please 
write your comments below, tear off at the dotted line and send to 

CIRCE - CMAS 

270 Education Building 


University of Illinois 
Urbana, Illinois 61801 
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Jacksonville Questionnaires 


MY HIGH SCHOOL 


Instructions: Given below are fifteen statements about the educational program 
at your high school. We all feel differently about such matters and this scale 
is an attempt to let you express your opinions. We are interested in how you 
feel the educational program at your high school has affected you. Respond to 
each item as follows: 


Agree Very Strongly: +3 Disagree Very Strongly: -3 
Agree Strongly: ee Disagree Strongly: -2 
Agree: +1 Disagree: -1 


For example, if you agree strongly with a statement, you would write 
a +2 on the line preceding the statement. If you disagree very strongly with a 
statement, you would write a -3 on the line preceding the statement. Respond 
to each statement as best you can. Go rapidly but carefully. Do not skip any 
items. 
1. I am satisfied with the education I am receiving. 
2. I have worked hard to get a good high school education. 


3. I prefer assignments in which the teacher tells me exactly what 
to do. 


4. I get satisfaction from the things I do in school. 


5. I study certain topics in greater depth than is required by my 
teachers. 


6. I have not been given enough responsibility in my high school 
education. 


7. I generally study for grades rather than for the sake of learning. 


8. The teachers are generally concerned about giving me a good 
education. 


9. When I have free time during the school day, I usually do other 
things instead of studying. 


10. I usually do most of my studying during the few days before an 
exam rather than throughout a unit of study. 


ll. The past school year was valuable in preparing me for the future. 


12. The teachers have given me responsibilities which most high school 
students don't have. 


13. The teachers put a lot of pressure on me. 
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The past school year was generally a waste of time for me. 


I study on my own topics which are not required by my teachers. 


freshman sophomore junior senior. 
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PARENT QUESTIONNAIRE 


Instructions: Given below are twelve statements about the individualized 
learning program at your son/daughter's high school. We all feel differently 
about such matters and this scale is an attempt to let you express your 
opinions. We are interested in how you feel the individualized learning 
program has affected your son/daughter who is participating in the program. 
Respond to each item as follows: 


Agree Very Strongly: +3 Disagree Very Strongly: ~3 
Agree Strongly: +2 Disagree Strongly: -2 
Agree: ot Disagree: -1 


For example, if you agree strongly with a statement, you would write 
a +2 on the line preceding the statement. If you disagree very strongly with a 
statement, you would write a -3 on the line preceding the statement. Respond to 
each statement as best you can. Please do not discuss any of the statements 
with you son/daughter before you complete the questionnaire. We would like to 
know your opinion. Do not skip any items. 


1. I am satisfied with the education my son/daughter is receiving. 
2. My son/daughter has worked hard to get a good high school education. 


3. My son/daughter gets satisfaction from the things he does in school. 


4. My son/daughter studies certain topics in greater depth than is 
required by his teachers. 


5. My son/daughter has not been given enough responsibility in his high 
school education. 


6. My son/daughter has been required to do too much studying. 


7. The individualized learning program has prepared my son/daughter 
well for his future. 


8. The teachers have given my son/daughter responsibilities which most 
high school students don't have. 


9. My son/daughter is proud to be in the individualized learning 
program. 


10. My son/daughter would learn more in a traditional high school 
program. 


11. My son/daughter has matured in assuming responsibility through his 
participation in the individualized learning program. 


12. I am generally familiar with the individualized learning progran. 
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CIRCE Resource Center Topical File 


Topics which are currently cross-referenced by card file in the 
CIRCE Resource Center include: 


Biological Science Curriculum Study 
Bureau of Educational Research 

Center for the Study of Evaluation 
Cooperative Educational Research Laboratory 
Curriculum Projects 

Disadvantaged 

EPLE 

Evaluation Reports and Plans 

Gifted Projects 

Individual School Programs 

Individual School Projects 

Learning Research and Development Center 
Math Programs 

National Assessment 

Newsletter File 

OSPI 

Professional Organizations 

Programmed Instruction 

Rocky Mountain Educational Laboratory 
ScIS 

Self-Instruction Programs 

Special Education 

Tests and Testing Manuals 

Titles I, II, III, and IV 
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